Bring CI back to all-working condition #5822

rwgk · 2025-09-02T17:08:44Z

Description

Fix "🐍 3 • windows-latest • mingw64" CI job

Apparently msys2/setup-msys2@v2 cannot be run twice anymore:
- https://github.com/pybind/pybind11/actions/runs/17394902023/job/49417376616?pr=5796

Run msys2/setup-msys2@v2
  with:
    msystem: mingw64
    install: mingw-w64-x86_64-python-numpy mingw-w64-x86_64-python-scipy mingw-w64-x86_64-eigen3
    path-type: minimal
    update: false
    pacboy: false
    release: true
    location: RUNNER_TEMP
    platform-check-severity: fatal
    cache: true
  env:
    PYTHONDEVMODE: 1
    PIP_BREAK_SYSTEM_PACKAGES: 1
    PIP_ONLY_BINARY: numpy
    FORCE_COLOR: 3
    PYTEST_TIMEOUT: 300
    VERBOSE: 1
    CMAKE_COLOR_DIAGNOSTICS: 1
    MSYSTEM: MINGW64
Error: Trying to install MSYS2 to D:\a\_temp\msys64 but that already exists, cannot continue.

test_gil_scoped.py xfail macOS free-threading related
Disable build-ios job in tests-cibw.yml (commit 0f5a29d)
Remove macos_brew_install_llvm job because it started failing, to reduce our maintenance overhead:

Failures are tracked here: Bring CI back to all-working condition #5822 (comment)

Suggested changelog entry:

Placeholder.

…2@v2 cannot be run twice anymore): https://github.com/pybind/pybind11/actions/runs/17394902023/job/49417376616?pr=5796 ``` Run msys2/setup-msys2@v2 with: msystem: mingw64 install: mingw-w64-x86_64-python-numpy mingw-w64-x86_64-python-scipy mingw-w64-x86_64-eigen3 path-type: minimal update: false pacboy: false release: true location: RUNNER_TEMP platform-check-severity: fatal cache: true env: PYTHONDEVMODE: 1 PIP_BREAK_SYSTEM_PACKAGES: 1 PIP_ONLY_BINARY: numpy FORCE_COLOR: 3 PYTEST_TIMEOUT: 300 VERBOSE: 1 CMAKE_COLOR_DIAGNOSTICS: 1 MSYSTEM: MINGW64 Error: Trying to install MSYS2 to D:\a\_temp\msys64 but that already exists, cannot continue. ```

…tion

rwgk · 2025-09-02T22:19:21Z

Hi @henryiii, could you please review? This should stop the frequent CI / 🐍 (macos-13, 3.13t, -DCMAKE_CXX_STANDARD=11) / 🧪 (pull_request) failures.

henryiii · 2025-09-02T22:47:02Z

tests/test_gil_scoped.py

+            if env.WIN and env.PYPY:
+                pytest.xfail("[TEST-GIL-SCOPED] Windows PyPy: " + msg)
+            elif env.MACOS and not env.sys_is_gil_enabled():
+                pytest.xfail("[TEST-GIL-SCOPED] macOS free-threading: " + msg)


This doesn't always fail, does it? So strict=False?

The strict flag only applies to @pytest.mark.xfail(...) (the decorator/mark form). Inline pytest.xfail() doesn't take strict at all — it immediately aborts the test and reports XFAIL. There’s no concept of XPASS in this case, so strict isn't relevant here.

To be sure I looked it up the good-old way:

https://docs.pytest.org/en/stable/how-to/skipping.html#xfail-mark-test-functions-as-expected-to-fail

Note that no other code is executed after the pytest.xfail() call, differently from the marker.

henryiii · 2025-09-02T22:47:18Z

tests/test_gil_scoped.py

-            if env.PYPY and env.WIN:
-                pytest.skip(msg)
+            if env.WIN and env.PYPY:
+                pytest.xfail("[TEST-GIL-SCOPED] Windows PyPy: " + msg)


This didn't run before, now it will. Guessing that's fine?

See other comment: Note that no other code is executed after the pytest.xfail() call, differently from the marker.

henryiii · 2025-09-02T22:53:53Z

.github/workflows/ci.yml

+            install_mingw64_only: ""
+          - sys: mingw64
+            env: x86_64
+            install_mingw64_only: |


I'd name this "extra_install", I think, since it's there to install extra stuff. And you can leave it off on the other job, variables default to empty.

Done: commit bb8a794

rwgk · 2025-09-02T23:30:35Z

Thanks @henryiii!

WDYT about disabling the two failing CIBW tests in this PR, too, until someone gets a chance to fix those?

CIBW / iOS wheel macos-13 (pull_request)
CIBW / iOS wheel macos-13 (pull_request)Failing after 19s

CIBW / iOS wheel macos-14 (pull_request)
CIBW / iOS wheel macos-14 (pull_request)Failing after 11s

rwgk · 2025-09-03T00:11:29Z

Ugh...

It is not working as intended:

🐍 (macos-13, 3.13t, -DCMAKE_CXX_STANDARD=11) / 🧪

            if process.exitcode is None:
                assert t_delta > 0.9 * timeout
                msg = "DEADLOCK, most likely, exactly what this test is meant to detect."
                if env.WIN and env.PYPY:
                    pytest.xfail("[TEST-GIL-SCOPED] Windows PyPy: " + msg)
                elif env.MACOS and not env.sys_is_gil_enabled():
                    pytest.xfail("[TEST-GIL-SCOPED] macOS free-threading: " + msg)
>               raise RuntimeError(msg)
E               RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

@henryiii I'm a bit lost, should I maybe checking for env.PY_GIL_DISABLED here?

IIUC, it's failing even though the GIL is enabled? This has been happening with macOS ever since I added this test, but only fairly infrequently.

rwgk · 2025-09-03T06:17:13Z

For easy future reference, the errors after commit bd3900e were:

CIBW / iOS wheel macos-13 (pull_request) Failing after 1m

CIBW / iOS wheel macos-14 (pull_request) Failing after 35s

macos-13:

  + python -m build /Users/runner/work/pybind11/pybind11/tests --wheel --outdir=/private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/cibw-run-2u0hmsfw/cp313-ios_x86_64_iphonesimulator/built_wheel
  * Creating isolated environment: venv+pip...
  * Installing packages in isolated environment:
    - scikit-build-core
  * Getting build dependencies for wheel...
  * Building wheel...
  2025-09-03 05:08:04,030 - scikit_build_core - INFO - RUN: /private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/cibw-run-2u0hmsfw/cp313-ios_x86_64_iphonesimulator/build/venv/cibw_xbuild_tools/cmake -E capabilities
  2025-09-03 05:08:04,053 - scikit_build_core - INFO - CMake version: 3.31.6
  *** scikit-build-core 0.11.6 using CMake 3.31.6 (wheel)
  2025-09-03 05:08:04,054 - scikit_build_core - INFO - Implementation: cpython ios on x86_64
  2025-09-03 05:08:04,055 - scikit_build_core - INFO - Build directory: /private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/tmpx_swhd3d/build
  *** Configuring CMake...
  2025-09-03 05:08:04,089 - scikit_build_core - INFO - RUN: /private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/cibw-run-2u0hmsfw/cp313-ios_x86_64_iphonesimulator/build/venv/cibw_xbuild_tools/ninja --version
  2025-09-03 05:08:04,094 - scikit_build_core - INFO - Ninja version: 1.13.1
  2025-09-03 05:08:04,097 - scikit_build_core - INFO - RUN: /private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/cibw-run-2u0hmsfw/cp313-ios_x86_64_iphonesimulator/build/venv/cibw_xbuild_tools/cmake -S. -B/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/tmpx_swhd3d/build -DCMAKE_BUILD_TYPE:STRING=Release -C/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/tmpx_swhd3d/build/CMakeInit.txt -DCMAKE_INSTALL_PREFIX=/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/tmpx_swhd3d/wheel/platlib -DCMAKE_MAKE_PROGRAM=/private/var/folders/vk/nx37ffx50hv5djclhltc26vw0000gn/T/cibw-run-2u0hmsfw/cp313-ios_x86_64_iphonesimulator/build/venv/cibw_xbuild_tools/ninja -DPYBIND11_FINDPYTHON=TRUE
  CMake Error: Could not find CMAKE_ROOT !!!
  CMake has most likely not been installed correctly.
  Modules directory not found in
  
  CMake Error: Error executing cmake::LoadCache(). Aborting.
  
  
  *** CMake configuration failed
  
  ERROR Backend subprocess exited when trying to invoke build_wheel

macos-14: very similar

This reverts commit bd3900e. See also: pybind#5822 (comment)

rwgk · 2025-09-03T07:20:11Z

This is flaking the second time (this PR):

CI / macos-13 • brew install llvm (pull_request) Failing after 10m (start time: Wed, 03 Sep 2025 06:38:28 GMT)

I'll try a rerun.

For completeness, the first flake was:

https://github.com/pybind/pybind11/actions/runs/17422859619/job/49464452403 (start time: Wed, 03 Sep 2025 04:21:04 GMT)

There was one success between the 1st and 2nd failure (https://github.com/pybind/pybind11/actions/runs/17423607165/job/49466476407, start time: Wed, 03 Sep 2025 05:07:40 GMT).

Edit: 3rd failure now: https://github.com/pybind/pybind11/actions/runs/17425182772/job/49473499749?pr=5822

Edit: I added that job with PR #4326 — initially to inform PR #4319, but then I decided to keep it, to be proactive about testing clang development versions. I'm not sure we still have the manpower to be proactive in this way.

Edit: 4th failure now: CI / macos-13 • brew install llvm (pull_request)

…uce our maintenance overhead: Failures tracked here: pybind#5822 (comment)

Replaced brew upgrade with brew install for cmake.

Uninstall cmake before installing the latest version due to GitHub's local tap changes.

.github/workflows/tests-cibw.yml

henryiii · 2025-09-03T14:02:22Z

GitHub is unpinning cmake in the next runner release: actions/runner-images#12935, so hopefully this entire step will be not needed. The target date is in around a week, I think.

rwgk · 2025-09-03T16:06:24Z

Thanks @henryiii, awesome that you could get the CIBW jobs working again with a one-line change!

I'll mark my calendar to harvest the TEST-GIL-SCOPED xfail tags from the CI logs in a month or so, to see where we stand.

rwgk requested a review from henryiii as a code owner September 2, 2025 17:08

rwgk added 2 commits September 2, 2025 13:41

Add pytest.xfail("[TEST-GIL-SCOPED] macOS free-threading...)

61d7577

Change env.SYS_IS_GIL_ENABLED constant to env.sys_is_gil_enabled func…

25b4c33

…tion

rwgk changed the title ~~Fix "🐍 3 • windows-latest • mingw64" CI job~~ Fix "🐍 3 • windows-latest • mingw64" CI job; test_gil_scoped.py xfail macOS free-threading Sep 2, 2025

henryiii reviewed Sep 2, 2025

View reviewed changes

Change install_mingw64_only → extra_install

bb8a794

rwgk added 2 commits September 2, 2025 21:16

Also xfail if macOS and PY_GIL_DISABLED, show SOABI

13c1c14

build-ios: brew upgrade|install cmake

bd3900e

rwgk added 2 commits September 2, 2025 23:18

Revert "build-ios: brew upgrade|install cmake"

bd75761

This reverts commit bd3900e. See also: pybind#5822 (comment)

Disable build-ios job in tests-cibw.yml

0f5a29d

Remove macos_brew_install_llvm job because it started failing, to red…

e70ac7f

…uce our maintenance overhead: Failures tracked here: pybind#5822 (comment)

rwgk changed the title ~~Fix "🐍 3 • windows-latest • mingw64" CI job; test_gil_scoped.py xfail macOS free-threading~~ Get CI back in all-working condition Sep 3, 2025

rwgk changed the title ~~Get CI back in all-working condition~~ Bring CI back to all-working condition Sep 3, 2025

henryiii added 2 commits September 3, 2025 09:54

Fix iOS build step for cmake installation

4de21d8

Replaced brew upgrade with brew install for cmake.

Update cmake installation steps in CI workflow

38e9fa7

Uninstall cmake before installing the latest version due to GitHub's local tap changes.

henryiii reviewed Sep 3, 2025

View reviewed changes

.github/workflows/tests-cibw.yml Outdated Show resolved Hide resolved

Update .github/workflows/tests-cibw.yml

26496bb

rwgk merged commit cd56888 into pybind:master Sep 3, 2025
85 checks passed

github-actions bot added the needs changelog Possibly needs a changelog entry label Sep 3, 2025

rwgk removed the needs changelog Possibly needs a changelog entry label Sep 3, 2025

rwgk deleted the fix-windows-latest-mingw64-ci-job branch September 3, 2025 17:21

This was referenced Sep 4, 2025

NOT FOR MERGE -- testing to see if windows-latest jobs are broken #5825

Closed

Fix crash in gil_scoped_acquire #5828

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bring CI back to all-working condition #5822

Bring CI back to all-working condition #5822

Uh oh!

rwgk commented Sep 2, 2025 •

edited

Loading

Uh oh!

rwgk commented Sep 2, 2025

Uh oh!

henryiii Sep 2, 2025

Uh oh!

rwgk Sep 2, 2025

Uh oh!

henryiii Sep 2, 2025

Uh oh!

rwgk Sep 2, 2025

Uh oh!

henryiii Sep 2, 2025

Uh oh!

rwgk Sep 2, 2025

Uh oh!

rwgk commented Sep 2, 2025

Uh oh!

rwgk commented Sep 3, 2025 •

edited

Loading

Uh oh!

rwgk commented Sep 3, 2025

Uh oh!

rwgk commented Sep 3, 2025 •

edited

Loading

Uh oh!

Uh oh!

henryiii commented Sep 3, 2025

Uh oh!

rwgk commented Sep 3, 2025

Uh oh!

Uh oh!

Uh oh!

Bring CI back to all-working condition #5822

Bring CI back to all-working condition #5822

Uh oh!

Conversation

rwgk commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Suggested changelog entry:

Uh oh!

rwgk commented Sep 2, 2025

Uh oh!

henryiii Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

rwgk Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

henryiii Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

rwgk Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

henryiii Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

rwgk Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

rwgk commented Sep 2, 2025

Uh oh!

rwgk commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rwgk commented Sep 3, 2025

Uh oh!

rwgk commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

henryiii commented Sep 3, 2025

Uh oh!

rwgk commented Sep 3, 2025

Uh oh!

Uh oh!

Uh oh!

rwgk commented Sep 2, 2025 •

edited

Loading

rwgk commented Sep 3, 2025 •

edited

Loading

rwgk commented Sep 3, 2025 •

edited

Loading